{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 大模型推理配置自动推荐\n",
    "\n",
    "千帆平台提供了众多模型，每个模型都配备了多样化的可调参数。然而默认参数往往不能适应所有场合，根据应用场景的不同，最合适的模型及其参数配置也各不相同。寻找最佳匹配通常意味着要人工进行大量尝试，这一过程对于每个新场景都需重复，既耗时又复杂。\n",
    "\n",
    "为解决此问题，我们的SDK引入了推理配置自动推荐功能。您只需提供目标场景的数据集和评价标准，并定义搜索空间，SDK便能自动为您推荐最优的模型及配置。此项功能保证了用户在不同场景下都能轻松定位到或优化出最理想的模型配置，最大限度地挖掘模型潜力，实现性能的极致提升。\n",
    "\n",
    "本指南将介绍如何利用SDK中的推理配置推荐功能，并展示其带来的显著性能提升。\n",
    "\n",
    "**注意**：这个功能需要千帆 SDK 版本 >= 0.3.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import qianfan\n",
    "\n",
    "os.environ[\"QIANFAN_ACCESS_KEY\"] = \"your_access_key\"\n",
    "os.environ[\"QIANFAN_SECRET_KEY\"] = \"your_secret_key\"\n",
    "# 由于后续在调优配置的过程中会并发请求模型，建议限制 QPS 和重试次数，避免调用失败\n",
    "os.environ[\"QIANFAN_QPS_LIMIT\"] = \"3\"\n",
    "os.environ[\"QIANFAN_LLM_API_RETRY_COUNT\"] = \"5\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备工作\n",
    "\n",
    "为了描述使用场景，需要先准备如下内容：\n",
    "\n",
    "- 数据集 Dataset：根据目标场景准备一定量的数据\n",
    "- 评估方式 Evaluator：根据目标场景，选择待优化的指标，并提供评估函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据集使用的是千帆 SDK 中提供的 `Dataset` 模块，可以直接加载本地的数据集文件，也可以使用平台上预置的或者自行上传的数据集，具体加载方式参考 [文档](https://github.com/baidubce/bce-qianfan-sdk/blob/main/docs/dataset.md)。\n",
    "\n",
    "这里我们以一个 **角色扮演** 数据集为例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [04-24 19:05:37] dataset.py:408 [t:140542390477056]: no data source was provided, construct\n",
      "[INFO] [04-24 19:05:37] dataset.py:276 [t:140542390477056]: construct a file data source from path: ./example.jsonl, with args: {'input_columns': ['prompt'], 'reference_column': 'response'}\n",
      "[INFO] [04-24 19:05:37] file.py:298 [t:140542390477056]: use format type FormatType.Jsonl\n",
      "[INFO] [04-24 19:05:37] utils.py:347 [t:140542390477056]: start to get memory_map from /root/.qianfan_cache/dataset/root/work/bce-qianfan-sdk/cookbook/autotuner/example.arrow\n",
      "[INFO] [04-24 19:05:37] utils.py:275 [t:140542390477056]: has got a memory-mapped table\n"
     ]
    }
   ],
   "source": [
    "from qianfan.dataset import Dataset\n",
    "\n",
    "dataset = Dataset.load(\n",
    "    data_file=\"./example.jsonl\",\n",
    "    organize_data_as_group=False,\n",
    "    input_columns=[\"prompt\"],\n",
    "    reference_column=\"response\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [04-24 19:05:38] dataset.py:994 [t:140542390477056]: list local dataset data by 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'姓名': '美杜莎', '性别': '女', '物种': '蛇人族', '工作': '女王', '昵称': '彩鳞', '居住地': '加玛帝国塔戈尔大沙漠', '恋爱状态': '已婚', '其他': '本体为七彩吞天蟒→九彩吞天蟒，伴生灵魂为六阶魔兽紫幽炎蛇', '人物经历': '美杜莎，蛇人族女王，与萧炎相遇后，从敌意转为爱意，成为其妻子，育有萧潇。曾帮助萧炎对抗万蝎门，守护炎盟和蛇人族。在萧炎闭关期间，美杜莎的修为提升至斗尊巅峰。后在魂殿和冥河盟的攻击中受重伤，但萧炎及时出关，击败敌人。美杜莎后继续提升实力，最终成为斗圣，并与萧炎一同前往大千世界。', '人物关系': {'萧炎': '丈夫', '萧潇': '女儿', '萧薰儿': '姐妹，萧炎的妻子之一', '花蛇儿': '蛇卫队长，忠诚于美杜莎', '海波东': '敌人，前加玛帝国十大强者之一', '小医仙': '盟友，与美杜莎关系由初期的芥蒂转为深厚的信任', '药尘': '盟友，曾与美杜莎达成协议，帮助其提升修为', '紫妍': '盟友，因特殊感应待美杜莎很好，帮助其找到炼制复魂丹所需药材'}, '人物性格': '高傲，心狠手辣，对族群极为重视，对萧炎和女儿萧潇充满爱意，经历多次转变后，对萧炎的情感复杂但深厚', '修为': '斗圣', '特殊能力': '七彩吞天蟒→九彩吞天蟒变化，紫幽炎蛇灵魂', '主要成就': '帮助萧炎建立炎盟，成为西北大陆前三强大联盟之一；在多次战斗中保护族人和盟友；最终成为斗圣，前往大千世界', '影响力': '塔戈尔大沙漠附近的几个帝国不敢轻易动战争，显示出其强大的威慑力和影响力。', '评价': ['古河评价美杜莎为杀人不眨眼的狠角色。', '药尘认为突破了斗皇界限的美杜莎女王潜力无限。']}\n",
      "现在请你扮演一个角色扮演专家。请你根据上述信息扮演美杜莎女王进行对话。\n",
      "\n",
      "好的！现在我来扮演美杜莎女王。\n",
      "萧炎：（脸色狂变，急喝）松嘴！\n",
      "美杜莎女王：（眸间噙着冷笑，掐着萧炎脖子的纤细玉手微微紧了紧）……\n",
      "萧炎：（赤红着眼睛，青色火焰升腾而起）松嘴！我知道，以你的实力，要杀我很简单，不过你现在却并未如此做，这可和你那凶名有些不相符啊……我想，你的实力，应该还未恢复吧？立刻放开，否则，今日即使拼了一条命，也要让得你重伤！\n",
      "美杜莎女王：（盯着萧炎，其中杀意凛然）……（缓缓开口，声音酥麻而慵懒）你是第一个敢出言威胁我的人。\n",
      "萧炎：（硬起脖子，干声道）你也是第一个咬我的女人。\n",
      "美杜莎女王：（伸出娇嫩的殷红舌头，轻轻地舔了舔红润嘴唇）先前的那股能量，应该是青莲地心火莲座中的莲子所凝吧？当初若非因为收取青莲地心火时，被弄得重伤，这种宝贝，我断然是不会遗留下半点。\n",
      "萧炎：（干笑了笑，没有回答，掌心中的青色火焰依然升腾着）……\n",
      "美杜莎女王：（淡淡地道）没想到，这到头来，最大的好处，竟然是被你给得到了去，那古河，恐怕是被气得不轻。\n",
      "萧炎：（嘿嘿笑道）嘿嘿，女王陛下不也是成功进化了么？你得到的好处，也不少了。\n",
      "美杜莎女王：（淡淡地道）为了这进化，当初我所受到的那种苦难，你又不是没看见。当时若非处在关键时刻，我会亲自出手把你宰了的。\n",
      "萧炎：（讪讪地笑了笑）嘿嘿。\n",
      "美杜莎女王：（纤指轻轻点向萧炎，脸颊上浮现一抹阴柔的气息）不过……人类，不得不说，你的胆子很大……你不仅敢独身闯进沙漠深处，更是还敢将我所进化的本体当成宠物来养，这份胆气，这么多年来，我还真的是头一次见到。\n",
      "萧炎：（耸了耸肩，无辜地道）我可没把你当成宠物养，只是它喜欢跟着我罢了。\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for item in dataset[0]:\n",
    "    print(item['prompt'])\n",
    "    print(item['response'][0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评估采用的 SDK 提供的 Evaluator 模块，基于 Evaluator 实现 evaluate 方法即可。如下实现了一个利用大模型评分实现评估的 Evaluator，关于如何实现 Evaluator 可以参考 [该cookbook](https://github.com/baidubce/bce-qianfan-sdk/blob/main/cookbook/evaluation/local_eval_with_qianfan.ipynb)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qianfan.evaluation.evaluator import LocalEvaluator\n",
    "from qianfan import ChatCompletion\n",
    "from qianfan.common.prompt.prompt import Prompt\n",
    "from qianfan.utils.pydantic import Field\n",
    "\n",
    "from typing import Optional, Union, Any, Dict, List\n",
    "import re\n",
    "import json\n",
    "\n",
    "class LocalJudgeEvaluator(LocalEvaluator):\n",
    "    model: Optional[ChatCompletion] = Field(default=None, description=\"model object\")\n",
    "    cache: Dict[str, Any] = Field(default={}, description=\"cache for evaluation\")\n",
    "    eval_prompt: Prompt = Field(\n",
    "        default=Prompt(\n",
    "            template=\"\"\"你需要扮演一个裁判的角色，对一段角色扮演的对话内容进行打分，你需要考虑这段文本中的角色沉浸度和对话文本的通畅程度。你可以根据以下规则来进行打分，你可以阐述你对打分标准的理解后再给出分数：\n",
    "                \"4\":完全可以扮演提问中的角色进行对话，回答完全符合角色口吻和身份，文本流畅语句通顺\n",
    "                \"3\":扮演了提问中正确的角色，回答完全符合角色口吻和身份，但文本不流畅或字数不满足要求\n",
    "                \"2\":扮演了提问中正确的角色，但是部分语句不符合角色口吻和身份，文本流畅语句通顺\n",
    "                \"1\":能够以角色的口吻和身份进行一部分对话，和角色设定有一定偏差，回答内容不流畅，或不满足文本字数要求\n",
    "                \"0\":扮演了错误的角色，没有扮演正确的角色，角色设定和提问设定差异极大，完全不满意\n",
    "                你的回答需要以json代码格式输出：\n",
    "                ```json\n",
    "                {\"modelA\": {\"justification\": \"此处阐述对打分标准的理解\", \"score\": \"此处填写打分结果\"}}\n",
    "                ```\n",
    "\n",
    "                现在你可以开始回答了：\n",
    "                问题：{{input}}\n",
    "                ---\n",
    "                modelA回答：{{output}}\n",
    "                ---\"\"\",\n",
    "            identifier=\"{{}}\",\n",
    "        ),\n",
    "        description=\"evaluation prompt\",\n",
    "    )\n",
    "\n",
    "    class Config:\n",
    "        arbitrary_types_allowed = True\n",
    "\n",
    "    def evaluate(\n",
    "        self, input: Union[str, List[Dict[str, Any]]], reference: str, output: str\n",
    "    ) -> Dict[str, Any]:\n",
    "        score = 0\n",
    "        try:\n",
    "            # 渲染评估用的 prompt，传入输入、模型输出和参考答案\n",
    "            p, _ = self.eval_prompt.render(\n",
    "                **{\n",
    "                    \"input\": \"\\n\".join([i[\"content\"] for i in input[1:]]),\n",
    "                    \"output\": output,\n",
    "                    \"expect\": reference,\n",
    "                }\n",
    "            )\n",
    "            # 利用 cache 避免对同一结果反复进行评估，提升效率\n",
    "            if p in self.cache:\n",
    "                model_output = self.cache[p]\n",
    "                score = float(model_output[\"modelA\"][\"score\"])\n",
    "            else:\n",
    "                # 请求模型进行评估\n",
    "                r = self.model.do(messages=[{\"role\": \"user\", \"content\": p}], temperature=0.01)\n",
    "                content = r[\"result\"]\n",
    "                model_output = content\n",
    "                # 提取出 json 格式的评估结果\n",
    "                regex = re.compile(\"\\`\\`\\`json(.*)\\`\\`\\`\", re.MULTILINE | re.DOTALL)\n",
    "    \n",
    "                u = regex.findall(content)\n",
    "    \n",
    "                if len(u) == 0:\n",
    "                    score = 0\n",
    "                else:\n",
    "                    model_output = json.loads(u[0])\n",
    "                    score = float(model_output[\"modelA\"][\"score\"])\n",
    "                    self.cache[p] = model_output\n",
    "        except Exception as e:\n",
    "            score = 0\n",
    "        # 返回评估结果，这里字段需与后续推荐配置时设定的评估字段一致\n",
    "        return {\"score\": score}\n",
    "\n",
    "    def summarize(self, metric_dataset: Dataset) -> Optional[Dict[str, Any]]:\n",
    "        score_sum = 0\n",
    "        count = 0\n",
    "\n",
    "        for line in metric_dataset.list():\n",
    "            score_sum += line[\"score\"]\n",
    "            count += 1\n",
    "\n",
    "        return {\"score\": score_sum / float(count)}\n",
    "\n",
    "local_evaluator = LocalJudgeEvaluator(\n",
    "    model=ChatCompletion(model=\"ERNIE-Bot-4\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在进行参数推荐之前，我们可以先使用默认参数的模型获取 baseline，方便后续评估参数效果。\n",
    "\n",
    "这里我们使用较为便宜的 ERNIE-Speed 和较贵但效果更好的 ERNIE-3.5 作为基准"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] [04-25 10:18:37] model.py:387 [t:140542390477056]: service type should be specified before exec\n",
      "[WARNING] [04-25 10:18:37] model.py:387 [t:140542390477056]: service type should be specified before exec\n",
      "[INFO] [04-25 10:18:37] evaluation_manager.py:480 [t:140542390477056]: start to inference in batch during evaluation\n",
      "[INFO] [04-25 10:18:37] dataset.py:994 [t:140539819177728]: list local dataset data by None\n",
      "[INFO] [04-25 10:18:37] dataset.py:994 [t:140540356048640]: list local dataset data by None\n",
      "[INFO] [04-25 10:18:43] base.py:92 [t:140539517171456]: All tasks finished, exeutor will be shutdown\n",
      "[INFO] [04-25 10:19:58] base.py:92 [t:140539450029824]: All tasks finished, exeutor will be shutdown\n",
      "[INFO] [04-25 10:19:58] evaluation_manager.py:504 [t:140542390477056]: start to evaluate llm 0\n",
      "[INFO] [04-25 10:21:08] evaluation_manager.py:504 [t:140542390477056]: start to evaluate llm 1\n",
      "[INFO] [04-25 10:22:13] evaluation_manager.py:532 [t:140542390477056]: start to merge evaluation result dataset\n"
     ]
    }
   ],
   "source": [
    "from qianfan.evaluation import EvaluationManager\n",
    "from qianfan.model import Service\n",
    "\n",
    "eb_speed_model = Service(model=\"ERNIE-Speed\")\n",
    "eb_35_model = Service(model=\"ERNIE-3.5-8K\")\n",
    "\n",
    "em = EvaluationManager(local_evaluators=[local_evaluator])\n",
    "result = em.eval([eb_speed_model, eb_35_model], dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'None_None_ERNIE-Speed': {'score': 2.3},\n",
       " 'None_None_ERNIE-3.5-8K': {'score': 2.95}}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们可以得到，默认参数下 ERNIE-Speed 和 ERNIE-3.5 的得分分别为 2.3 和 2.95。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获取推荐配置\n",
    "\n",
    "接下来我们可以使用 SDK 的模型配置推荐功能，通过调整参数来提高模型的效果。\n",
    "\n",
    "为了获取推荐配置，还需要设置一个超参搜索空间，千帆 SDK 提供了如下表示搜索空间的类：\n",
    "\n",
    "- `Uniform`：表示一个均匀分布的搜索空间，包含两个参数 `low` 和 `high`，分别表示下界和上界。\n",
    "- `Categorical`：表示一个离散的搜索空间，包含一个参数 `choices`，表示一组候选值。\n",
    "\n",
    "这里我们使用较为便宜的 ERNIE-Speed 和 ERNIE-Bot-turbo 作为待挑选的模型，并在整个 temperature 的取值范围内进行尝试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qianfan.autotuner.space import Uniform, Categorical\n",
    "\n",
    "search_space = {\n",
    "    \"temperature\": Uniform(0.01, 0.99),  # 设定temperature的范围\n",
    "    \"model\": Categorical([\"ERNIE-Speed\", \"ERNIE-Bot-turbo\"]),  # 设定model的取值范围\n",
    "    # 更多其他参数也可以按同样方式设定\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之后就可以执行推荐"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [04-24 19:20:37] launcher.py:108 [t:140542390477056]: turn 0 started...\n",
      "[INFO] [04-24 19:20:37] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.31496131457375937, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 19:20:37] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:22:33] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.31496131457375937, 'model': 'ERNIE-Speed'}, metrics: {'score': 3.015, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 46.98, 'avg_total_tokens': 740.48, 'avg_req_latency': 2.1625465251600327, 'avg_tokens_per_second': 342.41113029704786, 'avg_cost': 0.0031498400000000005, 'total_cost': 0.31498400000000004, 'success_rate': 1.0, 'total_time': 116.14484143257141}\n",
      "[INFO] [04-24 19:22:33] launcher.py:108 [t:140542390477056]: turn 1 started...\n",
      "[INFO] [04-24 19:22:33] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.4036019386971203, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 19:22:33] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:25:55] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.4036019386971203, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.52, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 112.39, 'avg_total_tokens': 867.19, 'avg_req_latency': 2.4468880319799062, 'avg_tokens_per_second': 354.40526442818503, 'avg_cost': 0.0029387399999999996, 'total_cost': 0.29387399999999997, 'success_rate': 1.0, 'total_time': 202.28806948661804}\n",
      "[INFO] [04-24 19:25:55] launcher.py:108 [t:140542390477056]: turn 2 started...\n",
      "[INFO] [04-24 19:25:55] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.16720055585301796, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 19:25:55] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:28:44] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.16720055585301796, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.47, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 122.1, 'avg_total_tokens': 876.9, 'avg_req_latency': 2.7196664212399626, 'avg_tokens_per_second': 322.42924836355473, 'avg_cost': 0.0029970000000000014, 'total_cost': 0.29970000000000013, 'success_rate': 1.0, 'total_time': 169.14291787147522}\n",
      "[INFO] [04-24 19:28:44] launcher.py:108 [t:140542390477056]: turn 3 started...\n",
      "[INFO] [04-24 19:28:44] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.6559287030793655, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 19:28:44] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:31:14] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.6559287030793655, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.86, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 46.34, 'avg_total_tokens': 739.84, 'avg_req_latency': 1.8474827565901069, 'avg_tokens_per_second': 400.4584060992918, 'avg_cost': 0.0031447199999999993, 'total_cost': 0.3144719999999999, 'success_rate': 1.0, 'total_time': 149.5975947380066}\n",
      "[INFO] [04-24 19:31:14] launcher.py:108 [t:140542390477056]: turn 4 started...\n",
      "[INFO] [04-24 19:31:14] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.518229696758757, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 19:31:14] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:33:26] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.518229696758757, 'model': 'ERNIE-Speed'}, metrics: {'score': 3.07, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 44.77, 'avg_total_tokens': 738.27, 'avg_req_latency': 1.7505615700800627, 'avg_tokens_per_second': 421.73323841802085, 'avg_cost': 0.0031321600000000015, 'total_cost': 0.31321600000000016, 'success_rate': 1.0, 'total_time': 132.07445192337036}\n",
      "[INFO] [04-24 19:33:26] launcher.py:108 [t:140542390477056]: turn 5 started...\n",
      "[INFO] [04-24 19:33:26] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.5029926419309916, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 19:33:26] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:35:20] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.5029926419309916, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.87, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 45.11, 'avg_total_tokens': 738.61, 'avg_req_latency': 1.747246960650009, 'avg_tokens_per_second': 422.7278779899683, 'avg_cost': 0.003134880000000001, 'total_cost': 0.3134880000000001, 'success_rate': 1.0, 'total_time': 113.81561374664307}\n",
      "[INFO] [04-24 19:35:20] launcher.py:108 [t:140542390477056]: turn 6 started...\n",
      "[INFO] [04-24 19:35:20] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.601681858384282, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 19:35:20] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:37:48] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.601681858384282, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.71, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 48.81, 'avg_total_tokens': 742.31, 'avg_req_latency': 1.9318500884798777, 'avg_tokens_per_second': 384.248241841635, 'avg_cost': 0.003164480000000001, 'total_cost': 0.3164480000000001, 'success_rate': 1.0, 'total_time': 147.96901512145996}\n",
      "[INFO] [04-24 19:37:48] launcher.py:108 [t:140542390477056]: turn 7 started...\n",
      "[INFO] [04-24 19:37:48] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.8828521471141014, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 19:37:48] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:41:06] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.8828521471141014, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.12, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 111.14, 'avg_total_tokens': 865.94, 'avg_req_latency': 2.4367797103298288, 'avg_tokens_per_second': 355.3624467280184, 'avg_cost': 0.002931240000000001, 'total_cost': 0.2931240000000001, 'success_rate': 1.0, 'total_time': 198.16581869125366}\n",
      "[INFO] [04-24 19:41:06] launcher.py:108 [t:140542390477056]: turn 8 started...\n",
      "[INFO] [04-24 19:41:06] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.40232265449161325, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 19:41:06] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:44:05] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.40232265449161325, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.31, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 98.1, 'avg_total_tokens': 852.9, 'avg_req_latency': 2.364468116320022, 'avg_tokens_per_second': 360.71537362382566, 'avg_cost': 0.002853000000000001, 'total_cost': 0.2853000000000001, 'success_rate': 1.0, 'total_time': 179.2184703350067}\n",
      "[INFO] [04-24 19:44:05] launcher.py:108 [t:140542390477056]: turn 9 started...\n",
      "[INFO] [04-24 19:44:05] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.8831853737639803, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 19:44:05] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:46:57] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.8831853737639803, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.26, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 55.49, 'avg_total_tokens': 748.99, 'avg_req_latency': 1.8754410246496627, 'avg_tokens_per_second': 399.367396871311, 'avg_cost': 0.003217920000000001, 'total_cost': 0.3217920000000001, 'success_rate': 1.0, 'total_time': 172.23393559455872}\n",
      "[INFO] [04-24 19:46:57] launcher.py:108 [t:140542390477056]: turn 10 started...\n",
      "[INFO] [04-24 19:46:57] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.21995595072189184, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 19:46:57] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:49:48] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.21995595072189184, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.65, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 120.63, 'avg_total_tokens': 875.43, 'avg_req_latency': 2.665976889850135, 'avg_tokens_per_second': 328.3711885624077, 'avg_cost': 0.002988180000000001, 'total_cost': 0.2988180000000001, 'success_rate': 1.0, 'total_time': 170.6479024887085}\n",
      "[INFO] [04-24 19:49:48] launcher.py:108 [t:140542390477056]: turn 11 started...\n",
      "[INFO] [04-24 19:49:48] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.5919699904289235, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 19:49:48] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:52:37] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.5919699904289235, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.51, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 105.3, 'avg_total_tokens': 860.1, 'avg_req_latency': 2.4076088940000044, 'avg_tokens_per_second': 357.24240849228164, 'avg_cost': 0.002896200000000001, 'total_cost': 0.2896200000000001, 'success_rate': 1.0, 'total_time': 168.7891285419464}\n",
      "[INFO] [04-24 19:52:37] launcher.py:108 [t:140542390477056]: turn 12 started...\n",
      "[INFO] [04-24 19:52:37] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.4176047258098396, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 19:52:37] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:54:49] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.4176047258098396, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.97, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 46.77, 'avg_total_tokens': 740.27, 'avg_req_latency': 1.8897913130202142, 'avg_tokens_per_second': 391.7205010414193, 'avg_cost': 0.0031481600000000005, 'total_cost': 0.31481600000000004, 'success_rate': 1.0, 'total_time': 132.43865966796875}\n",
      "[INFO] [04-24 19:54:49] launcher.py:108 [t:140542390477056]: turn 13 started...\n",
      "[INFO] [04-24 19:54:49] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.6572193324684913, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 19:54:49] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 19:57:51] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.6572193324684913, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.385, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 110.04, 'avg_total_tokens': 864.84, 'avg_req_latency': 2.49874210244976, 'avg_tokens_per_second': 346.1101484431359, 'avg_cost': 0.0029246400000000022, 'total_cost': 0.2924640000000002, 'success_rate': 1.0, 'total_time': 181.92341089248657}\n",
      "[INFO] [04-24 19:57:51] launcher.py:108 [t:140542390477056]: turn 14 started...\n",
      "[INFO] [04-24 19:57:51] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.7654242964057021, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 19:57:51] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 20:00:30] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.7654242964057021, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.6, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 46.88, 'avg_total_tokens': 740.38, 'avg_req_latency': 1.8225030058204357, 'avg_tokens_per_second': 406.2435000850401, 'avg_cost': 0.00314904, 'total_cost': 0.314904, 'success_rate': 1.0, 'total_time': 158.96834349632263}\n",
      "[INFO] [04-24 20:00:30] launcher.py:108 [t:140542390477056]: turn 15 started...\n",
      "[INFO] [04-24 20:00:30] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.07011654070806107, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 20:00:30] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 20:01:49] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.07011654070806107, 'model': 'ERNIE-Speed'}, metrics: {'score': 3.07, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 44.65, 'avg_total_tokens': 738.15, 'avg_req_latency': 1.6612444400295499, 'avg_tokens_per_second': 444.3355729075427, 'avg_cost': 0.003131200000000003, 'total_cost': 0.3131200000000003, 'success_rate': 1.0, 'total_time': 79.46442461013794}\n",
      "[INFO] [04-24 20:01:49] launcher.py:108 [t:140542390477056]: turn 16 started...\n",
      "[INFO] [04-24 20:01:49] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.10113858551004723, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 20:01:49] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 20:04:54] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.10113858551004723, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.55, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 133.64, 'avg_total_tokens': 888.44, 'avg_req_latency': 2.8740974726600688, 'avg_tokens_per_second': 309.1196483248428, 'avg_cost': 0.0030662400000000005, 'total_cost': 0.30662400000000006, 'success_rate': 1.0, 'total_time': 184.2768337726593}\n",
      "[INFO] [04-24 20:04:54] launcher.py:108 [t:140542390477056]: turn 17 started...\n",
      "[INFO] [04-24 20:04:54] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.6955109742460135, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 20:04:54] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 20:08:10] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.6955109742460135, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.395, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 108.25, 'avg_total_tokens': 863.05, 'avg_req_latency': 2.569956088690087, 'avg_tokens_per_second': 335.8228585298119, 'avg_cost': 0.002913899999999999, 'total_cost': 0.2913899999999999, 'success_rate': 1.0, 'total_time': 196.5271017551422}\n",
      "[INFO] [04-24 20:08:10] launcher.py:108 [t:140542390477056]: turn 18 started...\n",
      "[INFO] [04-24 20:08:10] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.8289072661571132, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-24 20:08:10] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 20:11:30] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.8289072661571132, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.25, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 113.9, 'avg_total_tokens': 868.7, 'avg_req_latency': 2.577946247330183, 'avg_tokens_per_second': 336.9736668868321, 'avg_cost': 0.0029478, 'total_cost': 0.29478, 'success_rate': 1.0, 'total_time': 199.8943543434143}\n",
      "[INFO] [04-24 20:11:30] launcher.py:108 [t:140542390477056]: turn 19 started...\n",
      "[INFO] [04-24 20:11:30] launcher.py:109 [t:140542390477056]: suggested config list: [{'temperature': 0.9601946893702433, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-24 20:11:30] dataset.py:994 [t:140542390477056]: list local dataset data by None\n",
      "[INFO] [04-24 20:14:30] launcher.py:114 [t:140542390477056]: config: {'temperature': 0.9601946893702433, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.37, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 61.46, 'avg_total_tokens': 754.96, 'avg_req_latency': 2.1469297925102, 'avg_tokens_per_second': 351.64633824252695, 'avg_cost': 0.003265680000000001, 'total_cost': 0.3265680000000001, 'success_rate': 1.0, 'total_time': 179.85176062583923}\n",
      "[INFO] [04-24 20:14:30] launcher.py:92 [t:140542390477056]: max turn reached: 20\n",
      "[INFO] [04-24 20:14:30] launcher.py:102 [t:140542390477056]: tuning finished!\n",
      "[INFO] [04-24 20:14:30] launcher.py:104 [t:140542390477056]: best config: {'temperature': 0.518229696758757, 'model': 'ERNIE-Speed'}\n"
     ]
    }
   ],
   "source": [
    "import qianfan.autotuner\n",
    "\n",
    "context = await qianfan.autotuner.run(\n",
    "    search_space=search_space,\n",
    "    dataset=dataset,\n",
    "    evaluator=local_evaluator,\n",
    "    # 以下均为可选参数\n",
    "    suggestor=\"random\",  # 搜索算法，目前仅支持 \"random\"，更多算法敬请期待\n",
    "    cost_budget=20,      # 设定整个流程的预算，达到预算则终止流程，单位为 “元”\n",
    "    metrics=\"score\",     # 设定评估指标字段，与 Evaluator 输出对应\n",
    "    mode=\"max\",          # 设定评估指标最大化还是最小化\n",
    "    repeat=5,            # 重复推理次数，用于减少大模型输出随机性对结果准确性的干扰\n",
    "    max_turn=20,         # 设定最大尝试次数\n",
    "    # max_time=3600,     # 设定最大尝试时间，单位为秒\n",
    "    log_dir= \"./log\",    # 日志目录\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回的结果是一个 `Context` 对象，其中包含了整个搜索过程的所有上下文信息，例如可以通过如下方式获得搜索的最佳参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'temperature': 0.518229696758757, 'model': 'ERNIE-Speed'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "context.best"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个参数可以直接用于推理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "好的，我来扮演一位热情洋溢的朋友。\n",
      "\n",
      "嘿，好久不见！你最近过得怎么样？有什么新鲜事情可以分享吗？\n"
     ]
    }
   ],
   "source": [
    "chat = qianfan.ChatCompletion().do(messages=[{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": \"请扮演一个角色，然后说一句话\"\n",
    "}], **context.best)\n",
    "\n",
    "print(chat['result'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "context 中也包含了整个过程中尝试的记录，可以获取某一轮某一组配置的评估结果等信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERNIE-Speed\t0.31496131457375937\t3.015\n",
      "ERNIE-Bot-turbo\t0.4036019386971203\t2.52\n",
      "ERNIE-Bot-turbo\t0.16720055585301796\t2.47\n",
      "ERNIE-Speed\t0.6559287030793655\t2.86\n",
      "ERNIE-Speed\t0.518229696758757\t3.07\n",
      "ERNIE-Speed\t0.5029926419309916\t2.87\n",
      "ERNIE-Speed\t0.601681858384282\t2.71\n",
      "ERNIE-Bot-turbo\t0.8828521471141014\t2.12\n",
      "ERNIE-Bot-turbo\t0.40232265449161325\t2.31\n",
      "ERNIE-Speed\t0.8831853737639803\t2.26\n",
      "ERNIE-Bot-turbo\t0.21995595072189184\t2.65\n",
      "ERNIE-Bot-turbo\t0.5919699904289235\t2.51\n",
      "ERNIE-Speed\t0.4176047258098396\t2.97\n",
      "ERNIE-Bot-turbo\t0.6572193324684913\t2.385\n",
      "ERNIE-Speed\t0.7654242964057021\t2.6\n",
      "ERNIE-Speed\t0.07011654070806107\t3.07\n",
      "ERNIE-Bot-turbo\t0.10113858551004723\t2.55\n",
      "ERNIE-Bot-turbo\t0.6955109742460135\t2.395\n",
      "ERNIE-Bot-turbo\t0.8289072661571132\t2.25\n",
      "ERNIE-Speed\t0.9601946893702433\t2.37\n"
     ]
    }
   ],
   "source": [
    "for turn in context.history:\n",
    "    for trial in turn:\n",
    "        metrics = trial.metrics\n",
    "        config = trial.config\n",
    "        print(\"{}\\t{}\\t{}\".format(config['model'], config['temperature'], metrics['score']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上面可以看到，搜索出的最佳配置 `{'temperature': 0.518229696758757, 'model': 'ERNIE-Speed'}` 对应的分数约为 3.07。\n",
    "\n",
    "相较而言，默认参数下 ERNIE-Speed 和 ERNIE-3.5 的得分分别为 2.3 和 2.95。\n",
    "\n",
    "可以看到推荐的参数配置 ERNIE Speed 表现相较于默认参数有了巨大提升，并且已经能够与默认参数的 ERNIE 3.5 所媲美，而与此同时 ERNIE Speed 的单价仅为 ERNIE 3.5 的三分之一，能够在大幅降低成本的前提下保证性能几乎与更大的模型持平，还能获得更高的 token 吞吐量，有助于在实际应用场景下实现降本增效。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py311",
   "language": "python",
   "name": "py311"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
